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GAO 

^^^^^^^Accountability  *  Integrity  *  ReliabHity _ 

United  States  General  Accounting  Office 
Washington,  DC  20548 


October  19,  2001 

The  Honorable  Christopher  Shays 
Chairman,  Subcommittee  on  National  Security, 

Veterans’  Affairs,  and  International  Relations 
Committee  on  Government  Reform 
House  of  Representatives 

Dear  Mr.  Chairman: 

The  Joint  Strike  Fighter  Program  is  the  most  expensive  aircraft  program  in 
the  Department  of  Defense  (DOD).  It  is  intended  to  produce  affordable, 
next-generation  aircraft  to  replace  aging  aircraft  in  the  military  services’ 
inventories.  As  currently  planned,  the  program  will  cost  about  $200  billion 
to  develop  and  procure  about  3,000  aircraft  and  related  support 
equipment.  'Two  contractor  teams,  led  by  the  Boeing  Company  and  by  the 
Lockheed  Martin  Aeronautics  Company,  are  competing  for  the  engineering 
and  manufacturing  development  phase.  This  phase  of  the  program  is 
projected  to  last  about  8  years  and  cost  about  $20  billion  and  typically 
involves  large,  fixed  investments  in  human  capital,  facilities,  and  materials. 

Last  year,  we  testified  and  reported  that  a  key  objective  of  the  program’s 
acquisition  strategy  is  affordability  and  that  a  part  of  that  strategy — 
entering  into  engineering  and  manufacturing  development  with  low 
technical  risk — ^would  not  be  achieved  because  technologies  critical  to 
meeting  the  program’s  cost  and  requirement  objectives  were  projected  to 
be  at  low  levels  of  technical  maturity  in  April  2001,  the  date  then 
scheduled  for  awarding  the  engineering  and  manufacturing  development 
contract.’  We  stated  that  the  program’s  approach  was  not  consistent  with 
best  practices  in  which  technologies  are  more  fully  developed  before 
proceeding  into  product  development.  Organizations  that  use  best 
practices  recognize  that  delaying  the  resolution  of  technology  problems 
until  product  development — analogous  to  the  engineering  and 
manufacturing  development  phase — can  result  in  at  least  a  ten-fold  cost 


Joint  Strike  Fighter  Acquisition:  Development  Schedule  Should  Be  Changed  to  Reduce 
Risks  (GAO/T-NSIAD-00-1.32,  Mar.  16,  20001,'  Joint  Strike  Fighter  Acquisition: 
Development  Schedule  Should  Be  Changed  to  Reduce  Risks  (GAO./NSIAD-00-74,  May  9, 
2000);  and  Defense  Acquisitions:  Decisions  on  the  Joint  Strike  Fighter  Will  Be  Critical 
for  Acquisition  Reform  (GAO/T-NSIAD-00-173,  May  10,  2000). 
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increase;  delaying  the  resolution  until  after  the  start  of  production  could 
increase  costs  by  a  hundred-fold. 

Because  of  concerns  about  the  adequacy  of  the  Joint  Strike  Filter’s  short 
take-off  and  vertical  landing  flight  test  program,  the  maturity  of  its  critical 
technologies,  and  other  factors,  the  Fiscal  Year  2001  National  Defense 
Authorization  Act  directed  that  the  contract  for  the  aircraft’s  engineering 
and  manufacturing  development  not  be  awarded  imtil  certain  criteria  were 
met.^  For  example,  the  act  required  that  the  program’s  short  take-off  and 
vertical  landing  demonstration  aircraft  acciunulate  at  least  20  horns  of 
flight  test  time  before  the  program  could  enter  the  engineering  and 
manufacturing  development  phase.  The  engineering  and  manufacturing 
development  contract  award  is  now  scheduled  for  October  2001. 

At  your  request,  we  assessed  whether  the  Joint  Strike  Fighter’s  critical 
technologies  will  have  matiued  to  a  low  technical  risk  at  the  start  of  the 
upcoming  engineering  and  manufacturing  development  phase.^  We 
assessed  technical  maturity  based  on  technology  readiness  levels — 
measures  pioneered  by  the  National  Aeronautics  and  Space 
Administration  and  adopted  by  the  Air  Force  Research  Laboratory  to 
determine  the  readiness  of  technologies  to  be  incorporated  into  a  weapon 
or  other  type  of  system.''  The  research  laboratoiy  considers  a  technology 
to  be  low  risk  for  the  engineering  and  manufacturing  development  stage 
when  a  prototype  of  that  technology  has  been  developed  that  includes  all 
of  its  critical  components  in  approximately  the  same  size,  weight,  and 
configuration  of  the  end  product  and  that  prototype  has  been 
demonstrated  to  work  in  an  environment  similar  to  the  planned 
operational  system. 


Results  in  Brief 


Although  the  Joint  Strike  Fighter  program  has  made  good  progress  in 
some  technology  areas,  the  program  is  at  risk  of  not  meeting  its 
affordability  objective  because  critical  technologies  are  not  projected  to 


^Section  212,  P.L.  106-398,  Oct.  30, 2000. 

technology  is  considered  to  be  mature  when  it  has  been  developed  to  a  point  that  it  can 
be  readily  integrated  into  a  new  product  and  counted  on  to  meet  product  requirements. 

^he  Air  Force  Research  Laboratory  is  a  science  and  technology  organization  that  matures 
advanced  technologies  to  the  point  that  they  can  be  included  in  weapon  system  programs 
and  be  expected  to  perform  as  required. 
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be  matured  to  levels  that  we  believe  would  indicate  a  low  risk  program  at 
the  planned  start  of  engineering  and  manufacturing  development  in 
October  2001.  In  fact,  the  critical  technologies  are  projected  to  be  at  the 
same  level  of  maturity  in  October  2001  as  they  were  projected  to  be  in 
April  2001,  the  previous  scheduled  date  for  the  engineering  and 
manufacturing  development  contract.  Several  of  these  technologies  xmder 
development  are  critical  to  making  the  long-term  ownership  costs  of  the 
program  affordable.  These  technologies  remain  at  higher  risk  levels  for 
engineering  and  manufacturing  development  because  (1)  they  have  not 
been  developed  to  approximately  the  same  size,  weight,  and  configuration 
of  the  end  product  and/or  (2)  they  have  not  been  demonstrated  to  work  in 
an  environment  similar  to  the  planned  operational  system.  For  example, 
the  contractors  have  demonstrated  certain  key  component  technologies  in 
flight  environments,  but  the  tested  hardware  was  not  always  the  same  size 
and  weight  required  for  the  Joint  Strike  Fighter  aircraft.  In  other  instances, 
components  built  to  the  required  size  and  weight  were  only  demonstrated 
in  grovmd-tests.® 

Based  on  oin-  work  on  best  practices  in  product  development,  and  the 
importance  of  the  Joint  Strike  Fighter  to  DOD  and  the  industrial  base,  we 
believe  that  DOD  needs  to  ensure  that  the  program’s  critical  technologies 
are  at  demonstrated  levels  of  maturity  before  making  engineering  and 
manufacturing  investments  in  the  program.  Failure  to  do  so  could  result  in 
increases  in  both  the  production  and  long-term  ownership  costs,  schedule 
delays,  and  compromised  performance  as  problems  arise  in  product 
development.  Moreover,  the  impact  of  failing  to  mature  one  critical 
technology  could  ripple  throughout  the  program.  Therefore,  we  are 
recommending  that  DOD  ensure  that  critical  technologies  are  mature 
before  proceeding  into  engineering  and  manufacturing  to  improve  the 
likelihood  of  meeting  program  expectations  or  to  take  additional  actions  if 
it  chooses  to  accept  the  risk  of  immature  technologies. 

In  commenting  on  our  report,  DOD  said  that  it  assessed  the  technology 
maturity  of  the  Joint  Strike  Fighter  to  be  sufficient  for  the  next  phase  of 
the  program.  DOD  also  said  that  the  Joint  Strike  Fighter  Program  Office 
has  implemented  a  rigorous  risk  management  program  that  will  continue 
to  monitor  and  address  technology  risks,  as  well  as  other  risks,  throughout 
the  program’s  life.  We  disagree  with  DOD’s  assertion  that  technology  is 


“Due  to  the  current  Joint  Strike  Fighter  competition,  the  technologies  are  not  specified  so 
as  to  not  associate  them  with  either  Boeing  or  Lockheed  Martin,  respectively. 
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mature  enough  to  move  forward.  The  technology  readiness  level 
assessment  conducted  as  part  of  our  review  of  the  Joint  Strike  Fighter 
showed  that  critical  technologies  are  not  projected  to  be  matured  to  levels 
that  would  stem  risks  at  the  start  of  engineering  and  manufacturing 
development.  Our  previous  work  has  shown  that  when  programs  proceed 
in  this  fashion,  they  experience  delays,  rework,  and  substantial  cost 
increases  that  could  force  the  Department  to  divert  much-needed  funds 
from  other  important  weapon  system  programs. 


Background 


The  Joint  Strike  Fighter  Program  is  structured  to  use  a  common 
production  line  to  produce  three  versions  of  a  single  aircraft.  These 
aircraft  will  be  tailored  to  meet  conventional  flight  requirements  for  the 
U.S.  Air  Force,  short  take-off  and  vertical  landing  characteristics  for  the 
U.S.  Marine  Corps,  and  carrier  operation  suitability  needs  for  the  U.S. 
Navy.  The  program  will  also  provide  aircraft  to  the  British  Royal  Navy  and 
Air  Force.  Table  1  shows  the  services’  planned  use  for  the  Joint  Strike 
Fighter. 


Table  1:  Military  Services’  Planned  Use  for  the  Joint  Strike  Fighter 

Service 

Quantity 

Planned  use 

U.S.  Air  Force 

1,763 

Replacement  for  F-1 6  and  A-1 0;  complement  the  F-22 

U.S.  Marine  Corps 

609 

Replacement  for  the  AV-8B  and  F/A-18  C/D 

U.S.  Navy 

480 

Complement  the  F/A-18  E/F 

Great  Britain 

150 

Replacement  for  the  Sea  Harrier  and  GR.7 

Source:  Joint  Strike  Fighter  program  office. 


A  key  objective  of  the  Joint  Strike  Fighter  acquisition  strategy  is 
affordability — ^reducing  the  development,  production,  and  ownership  costs 
of  the  program  relative  to  prior  fighter  aircraft  programs.  To  achieve  its 
affordability  objective,  the  Joint  Strike  Fighter  program  has  incorporated 
various  acquisition  initiatives  into  the  program’s  acquisition  strategy  and 
various  technological  advances  into  the  fighter.  Among  the  acquisition 
initiatives  planned  was  to  develop  critical  technologies  to  a  level  where 
they  represent  low  technical  risk  before  the  engineering  and 
manufacturing  contract  is  awarded.  The  expectation  was  that 
incoiporating  these  initiatives  into  the  acquisition  strategy  would  avoid 
cost  growth,  schedule  slippage,  and  performance  shortfalls  that  have  been 
experienced  in  other  weapon  acquisition  programs. 
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To  date,  the  Joint  Strike  Fighter  Program  has  awarded  contracts  totaling 
over  $2  billion  to  Boeing  and  Lockheed  Martin  for  the  current  concept 
demonstration  phase.  During  this  phase,  DOD  required  each  contractor  to 
design  and  build  two  aircraft  to  demonstrate  the  following: 

•  commonality/modularity  to  validate  the  contractors’  ability  to  produce 
three  aircraft  versions  on  the  same  production  line; 

•  the  aircraft’s  ability  to  do  a  short  take-off  and  vertical  landing,  hover,  and 
transition  to  forward  flight;  and 

•  satisfactory  low  airspeed,  carrier  approach  flying  and  handling  qualities. 

Each  contractor  was  required  to  submit  a  Preferred  Weapon  System 
Concept,  which  outlines  its  final  design  concept  for  developing  a  Joint 
Strike  Fighter  aircraft  that  is  affordable  and  meets  performance 
requirements.  The  Preferred  Weapon  System  Concept  includes  results 
from  the  flight  and  ground  demonstrations  and  is  being  used  by  DOD  to 
select  the  winning  aircraft  design  and  to  award  the  engineering  and 
manufacturing  development  contract.  During  engineering  and 
manufacturing  development,  the  Joint  Strike  Fighter  will  be  fully 
developed,  engineered,  designed,  fabricated,  tested,  and  evaluated  to 
demonstrate  that  the  production  aircraft  will  meet  stated  requirements. 
Critical  junctures  in  engineering  and  manufacturing  development  are  the 
preliminary  and  critical  design  reviews  and  commitments;  testing  of 
aircraft;  and  commitments  to  production  hardware,  including  the  purchase 
of  long  lead  production  items.  It  is  at  the  critical  design  review  that 
decisions  are  made  toward  finalizing  the  aircraft  design  and  begin  building 
test  aircraft.  About  two-thirds  of  engineering  and  manufacturing 
development  funding  will  be  spent  after  this  review.  Figure  1  shows 
planned  Joint  Strike  Fighter  aircraft  designs  by  contractor. 
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Technology  Readiness 
Assessments  Provide 
Opportunities  to 
Improve  Outcomes 


In  our  previous  work  on  best  business  practices,  commercial  firms  have 
told  us  that  a  key  part  of  product  development  is  getting  the  technology 
into  the  right  size,  weight,  and  configuration  needed  for  the  intended 
product — in  this  case,  the  final  Joint  Strike  Fighter  design.  Once  this  has 
been  demonstrated,  the  technology  is  at  an  acceptable  level  for  product 
development.  Technology  readiness  levels  (TRL)  can  be  used  to  assess  the 
maturity  of  technology  and  can  reveal  whether  a  gap  exists  between  a 
technology’s  maturity  and  the  maturity  demanded  for  successful  inclusion 
in  the  intended  product.  Defining  this  gap  for  the  Joint  Strike  Fighter 
technologies  is  important  for  determining  whether  they  can  be  expected  to 
demonstrate  required  capabilities  before  being  integrated  into  the  aircraft 
design.  Readiness  levels  are  measured  along  a  scale  of  one  to  nine,  starting 
with  paper  studies  of  the  basic  concept,  proceeding  with  laboratory 
demonstrations,  and  ending  with  a  technology  that  has  proven  itself  on  the 
intended  product.  (See  app.  I  for  a  detailed  description  of  TRLs.)  The  Air 
Force  Research  Laboratory  considers  TRL  7  an  acceptable  risk  for  starting 
the  engineering  and  manufacturing  development  phase.  The  readiness 
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level  definitions  state  that  for  a  technology  to  be  rated  at  TRL  7,  it  must  be 
demonstrated  using  prototype  hardware  (such  as  a  complete  radar 
subsystem)  that  is  the  same  size,  weight,  and  configuration  as  that  called 
for  in  the  final  aircraft  design  and  that  prototype  has  to  be  demonstrated 
to  work  in  an  environment  similar  to  the  planned  operational  system. 

We  have  previously  reviewed  the  impact  of  incorporating  technologies 
into  new  product  and  weapon  system  designs.®  The  results  showed  that 
programs  met  product  objectives  when  the  technologies  were  matured  to 
higher  levels  and  conversely  showed  that  cost  and  schedule  problems 
arose  when  programs  started  when  technologies  were  at  low  readiness 
levels.  For  example,  the  Joint  Direct  Attack  Munition  (JDAM)  used 
modified  variants  of  proven  components  for  guidance  and  global 
positioning.  It  also  used  mature,  existing  components  from  other  proven 
manufacturing  processes  for  its  own  system  for  controlling  tail  fin 
movements.  The  munition  was  touted  for  its  performance  in  Kosovo  and 
was  purchased  for  less  than  half  of  its  expected  unit  cost.  However,  the 
Comanche  helicopter  program  began  with  critical  technologies  such  as  the 
engine,  rotor,  and  integrated  avionics  at  TRL  levels  of  5  or  below.  That 
program  has  seen  101  percent  cost  growth  and  120-percent  schedule 
slippage  as  a  result  of  these  low  maturity  levels  and  other  factors. 

In  commenting  on  our  report  concerning  better  management  of 
technology  development,  DOD  agreed  that  TRLs  are  important  and 
necessary  in  assisting  decision  makers  in  deciding  on  when  and  where  to 
insert  new  technologies  into  weapons  system  programs  and  that  it  is 
desirable  to  mature  technologies  to  TRL  7  prior  to  entering  the  engineering 
and  manufacturing  development  phase  of  a  weapon  system  program.’ 

Since  that  time,  DOD  has  adopted  the  technology  readiness  levels  as  a 
means  of  assessing  the  technological  maturity  of  new  mgjor  programs.  In  a 
July  6,  2001,  memorandum,  the  Deputy  Under  Secretary  of  Defense 
(Science  and  Technology)  stated  that  new  DOD  regulations  require  that 
the  military  services’  science  and  technology  executives  conduct  a 
technology  readiness  level  assessment  for  critical  technologies  identified 
in  major  weapon  systems  programs  prior  to  the  start  of  engineering  and 


^Best  Practices:  Successful  Application  to  Weapon  Acquisitions  Requires  Changes  in 
DOD’s  Environment  (GAO/NSIAD-98-56,  Feb.  24, 1998)  and  Best  Practices:  Better 
Management  of  Technology  Development  Can  Improve  Weapon  System  Outcomes 
(GAO/NSlAD-99-162,  JulySO,  1999). 

’GAO/NSIAD-99-162,  July  30, 1999. 
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manufacturing  development  and  production.  The  memorandum  notes  that 
technology  readiness  levels  are  the  preferred  approach  for  all  new  major 
programs  unless  the  Deputy  Under  Secretary  approves  an  equivalent 
assessment  method. 

The  Joint  Strike  Fighter  Program,  like  many  other  DOD  programs,  has 
used  risk  management  plans  and  engineering  judgment  as  a  way  of 
assessing  technological  maturity.  The  Principal  Deputy  Under  Secretary  of 
Defense  (Acquisition  and  Technology)  has  determined  that  these  means 
will  continue  to  be  used  by  DOD  and  the  Joint  Strike  Fighter  contractors 
to  assess  the  program’s  technological  risk.  Risk  management  plans  and 
judgment  are  necessary  to  managing  any  major  development  effort  like  the 
Joint  Strike  Fighter.  However,  without  an  imdeipinning  such  as 
technology  readiness  levels  that  allow  transparency  into  program 
decisions,  these  methods  allow  significant  technical  unknowns  to  be 
judged  acceptable  risks  because  a  plan  exists  for  resolving  the  unknowns 
in  the  future.  Experience  on  previous  programs  has  shown  that  such 
methods  have  rarely  assessed  technical  unknowns  as  a  high  or 
imacceptable  risk;  consequently,  they  failed  to  guide  programs  to  meet 
promised  outcomes.  Technology  readiness  levels  are  based  on  actual 
demonstrations  of  how  well  technologies  actually  perform.  Their  strength 
lies  in  the  fact  that  they  characterize  knowledge  that  exists  rather  than 
plans  to  gain  knowledge  in  the  future;  they  are,  thus,  less  susceptible  to 
optimism. 


Critical  Technologies 
Are  Not  Projected  to 
Be  at  Low  Risk 


In  May  2000  we  reported  that  all  of  the  eight  technologies  identified  by  the 
Joint  Strike  Fighter  program  office  as  critical  to  the  program  were 
expected  to  be  at  maturity  levels  below  that  considered  acceptable  for  low 
risk  when  entering  engineering  and  manufacturing  development  (TRL  7). 
The  eight  critical  technologies  are:  prognostics  and  health  management, 
integrated  fli^t  propulsion  control,  subsystems,  integrated  support 
system,  integrated  core  processor,  radar,  manufacturing,  and  mission 
systems  integration.  (See  app.  II  for  a  description  of  these  technologies.) 
During  our  review  last  year,  we  worked  with  the  two  competing 
contractors  and  the  program  office  to  arrive  at  the  applicable  TRLs  for  the 
critical  technologies.  Specifically,  on  separate  visits  to  the  contractors, 
with  program  office  personnel  present,  we  asked  the  contractors’  relevant 
technology  managers  to  score  the  technologies  they  considered  critical  to 
enable  their  Joint  Strike  Fighter  design  to  meet  DOD  requirements  for  the 
aircraft.  At  that  time,  we  also  asked  them  to  describe  their  plans  to  mature 
the  technologies  to  the  planned  start  of  the  engineering  and  manufacturing 
development  phase,  then  scheduled  for  April  2001.  Upon  reviewing  these 
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Critical  Technology  Areas 


scores  with  the  program  office  and  in  order  to  gain  an  overall  Joint  Strike 
Fighter  Program  perspective  on  technical  maturity,  the  Joint  Strike  Fighter 
office  agreed  to  provide  us  with  TRL  scores  for  the  eight  technologies  they 
considered  critical  for  meeting  program  cost  and  performance 
requirements.  Figure  2  reflects  the  program  office  scores  at  the  time  of  our 
last  review.  Due  to  the  current  Joint  Strike  Fighter  competition,  the 
specific  technologies  mentioned  previously  are  not  linked  to  scores  so  as 
not  to  divulge  competition  sensitive  information. 


Figure  2:  Actual  and  Projected  Joint  Strike  Fighter  Criticai  Technology  Readiness  Levels 


Acceptable  risk  for  program  start 


Acceptable  risk  for  entry  into 
engineering  and  manufacturing 
development  phase 


Technology  1 
Technology  2 
Technology  3 
Technology  4 
Technology  5 
Technology  6 
Technology  7 
Technology  8 

01  23456789 

Technology  Readiness  Level 

B  Program  Start  (contract  award  - 1 1/96) 

E  Previous  GAO  Review  (12/99) 

□  Prior  Planned  Engineering  and  Manufacturing  Development  Start  (4/01) 

Source:  Joint  Strike  Fighter  program  office. 

As  the  figure  shows,  all  eight  technologies  were  projected  to  be  below  the 
level  of  maturity  (TRL  7)  considered  acceptable  for  low  risk  when  entering 
the  engineering  and  manufacturing  development  phase  and  six  of  the 
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technologies  were  projected  to  be  below  the  level  of  maturity  (TRL  6)  that 
is  considered  low  risk  for  entering  the  demonstration  phase,  which  the 
Joint  Strike  Fighter  Program  began  in  1996. 

During  our  current  review,  we  again  visited  the  two  competing  contractors 
to  discuss  the  status  of  the  eight  technologies.  We  learned  that  they  have 
essentially  accomplished,  or  plan  to  accomplish  by  October  2001,  the 
technology  development  and  demonstrations  that  they  planned  to 
accomplish  as  of  April  2001.  Thus  figure  2  represents  the  current 
assessment  of  technical  maturity.  While  two  of  these  areas  are  very  close 
to  appropriate  maturity  levels,  the  Joint  Strike  Fighter’s  critical 
technologies  are  not  projected  to  be  matured  to  levels  that  we  believe 
would  indicate  a  low  risk  program  at  the  planned  start  of  the  engineering 
and  manufacturing  development  phase.  Key  component  technologies 
remain  at  higher  risk  levels  for  engineering  and  manufacturing 
development  because  (1)  they  have  not  been  developed  to  approximately 
the  same  size,  weight,  and  configuration  called  for  in  the  final  aircraft 
design  and/or  (2)  they  have  not  been  demonstrated  to  work  in  an 
environment  similar  to  the  planned  operational  system. 

The  Joint  Strike  Fighter  Program  has  made  good  progress  in  some 
technology  areas.  For  example,  contractor  and  program  officials  told  us 
that  because  of  concerns  about  propulsion  technology,  both  contractors 
focused  considerable  attention  on  that  area.  Both  contractors  flew  aircraft 
that  demonstrated  the  capability  for  short  take-off  and  vertical  landing  and 
accumulated  at  least  20  hours  of  flight  time  on  those  aircraft,  which  should 
satisfy  the  requirement  in  the  Fiscal  Year  2001  National  Defense 
Authorization  Act.  In  some  other  areas,  the  technology  maturation  has  not 
been  uniform  across  all  critical  components  of  a  technology.  For  example, 
the  radar  has  a  number  of  critical  components  that  must  work  together  as 
a  system.  Both  contractors  have  made  considerable  progress  on  one  or 
more  of  those  components,  but  the  other  critical  components  have  not 
been  matured  to  an  acceptable  level  of  risk.  In  order  for  this  technology  to 
achieve  a  TRL  level  of  7,  all  components  had  to  be  (1)  demonstrated  in  the 
size  and  weight  required  to  meet  aircraft  capabilities,  (2)  integrated 
together  as  they  would  be  in  the  final  aircraft  design,  and  (3)  flown  in  an 
environment  similar  to  what  the  Joint  Strike  Fighter  will  be  subjected.  To 
demonstrate  some  critical  technologies,  both  contractors  flew  key 
electronic  and  other  components  in  flying  avionics  test  beds  (commercial 
aircraft  reconfigured  as  flying  laboratories).  While  these  tests  occurred  in 
a  relevant  environment  (e.g.,  in  flight),  the  tested  hardware  was  not  always 
the  same  size  and  weight  required  for  the  Joint  Strike  Fighter  aircraft. 
Conversely,  some  components  were  built  to  the  required  size  and  weight. 
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but  were  demonstrated  only  in  ground-testing  environments.  By  not 
having  matured  all  critical  technology  areas  to  appropriate  maturity  levels, 
the  program  remains  at  risk  for  achieving  cost  and  performance  goals 
upon  entering  product  development. 


Conclusions 


Moving  into  engineering  and  manufacturing  development  creates  an 
expectation  that  the  Joint  Strike  Fighter  can  be  delivered  for  a  stated  time 
and  dollar  investment  and  with  a  given  set  of  capabilities.  The  decisions 
the  Department  of  Defense  makes  now  and  over  the  next  2  years  will 
largely  determine  whether  those  expectations  can  be  met. 

A  key  component  of  the  Joint  Strike  Fighter  Program’s  acquisition  strategy 
is  to  enter  the  engineering  and  manufacturing  development  phase  with  low 
technical  risk.  The  program  will  not  have  achieved  that  point  by  October 
2001  because  technologies,  which  the  Joint  Strike  Fighter  Program  Office 
identified  as  critical  to  meeting  the  program’s  cost  and  requirements 
objectives,  will  not  have  been  matured  to  an  acceptable  risk  level.  By 
entering  the  engineering  and  manufacturing  development  phase  with 
immature  critical  technologies,  the  program  will  need  to  continue  to 
develop  those  technologies  at  the  same  time  it  will  be  concentrating  on 
production  issues  and  the  integration  of  subsystems  into  a  Joint  Strike 
Fighter.  This  approach  would  not  be  consistent  with  best  practices.  In  fact, 
it  would  more  closely  follow  DOD’s  traditional  practices  in  weapon  system 
programs  that  have  often  resulted  in  cost  increases,  schedule  delays,  and 
compromised  performance. 


R6COnUT16n(i3.tionS  for  eliminate  one  of  the  major  sources  of  cost  and  schedule  risk,  we 

recommend  that  the  Secretary  of  Defense  delay  the  start  of  engineering 
Executive  Action  and  manufacturing  development  xmtil  critical  technologies  are  matured  to 

acceptable  levels. 

Alternatively,  if  the  Secretary  of  Defense  decides  to  accept  these  risks  and 
move  the  program  into  engineering  and  manufacturing  development  as 
scheduled,  we  recommend  that  the  Secretary  dedicate  the  resources  to 
ensuring  that  maturity  of  the  critical  technologies  is  demonstrated  by  the 
critical  design  review  or  defer  the  inclusion  of  immature  technologies 
from  the  approved  design. 
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Agency  Comments 
and  Our  Evaluation 


In  written  comments  on  a  draft  of  this  report,  the  Director  of  Strategic  and 
Tactical  Systems,  within  the  Office  of  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics,  partially  concurred  with  our 
recommendation.  DOD  contended  that  an  independent  technology 
readiness  assessment  it  carried  out  on  the  program  showed  that 
technology  has  extensively  matured  and  the  program  is  now  ready  to  enter 
into  systems  development  and  demonstration.  DOD  also  stated  that  the 
Joint  Strike  Fighter  Program  Office  has  implemented  a  risk  management 
program  that  will  continue  to  monitor  and  address  technology  risks,  as 
well  as  other  risks,  throughout  the  program’s  life.  The  full  text  of  DOD’s 
comments  is  included  in  appendix  III. 

We  disagree  with  the  Department’s  assessment  of  technological  maturity. 
The  TRL  assessment  conducted  as  part  of  our  review  showed  that 
technologies  critical  to  the  Joint  Strike  Fighter  Program  are  not  projected 
to  be  matured  to  levels  that  we  believe  would  indicate  a  low  risk  program 
at  the  planned  start  of  the  engineering  and  manufacturing  development 
phase.  Many  of  the  technologies  have  not  been  demonstrated  in  their 
appropriate  size  and  weight,  nor  have  they  been  demonstated  to  function 
in  an  environment  in  which  they  will  be  used.  For  example,  many  of  the 
technologies  are  still  in  the  laboratory  and  will  require  considerable 
maturation  before  they  can  be  incorporated  into  the  final  design.  By 
entering  the  engineering  and  manufacturing  development  phase  with 
immature  critical  technologies,  the  program  will  need  to  continue  to 
develop  those  technologies  at  the  same  time  it  will  be  concentrating  on 
engineering,  designing,  and  fabricating  the  product.  As  it  has  with  many 
other  DOD  programs,  this  approach  increases  the  likelihood  of  schedule 
delays  and  program  cost  increases.  This  is  primarily  why  DOD’s  new 
acquisition  regulations  emphasize  separating  technology  development 
from  product  development.  In  fact,  experience  has  shown  that  resolving 
technology  problems  in  product  development  can  result  in  at  least  a  ten¬ 
fold  cost  increase. 

Moreover,  DOD  incorrectly  states  that  the  tools  it  used  to  assess  its 
technology  and  the  TRLs  used  for  our  review  are  equivalent 
methodologies  for  assessing  technological  maturity.  The  Willoughby 
Templates  used  by  DOD  are  a  risk  management  tool.  They  can  be  an 
excellent  way  to  manage  program  risks,  but  in  practice  they  have  not  been 
used  to  identify  risk.  Identifying  risk  is  the  first  step  to  managing  it.  By 
contrast,  by  focusing  specifically  on  assessing  technology  maturity  against 
objective  standards,  TRLs  have  proven  successful  at  identifying  risks.  A 
more  appropriate  approach  for  DOD  to  take  is  to  use  technology  readiness 
levels  in  conjimction  with  a  management  tool  such  as  the  Willoughby 
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templates  since  this  can  result  in  more  informed  decision  making  and 
fewer  imanticipated  problems  in  an  acquisition  program.  In  fact,  the  Joint 
Strike  Fighter  program  provides  DOD  with  an  excellent  opportunity  to 
apply  these  concepts  in  tandem. 


Scope  and 
Methodology 


To  assess  whether  the  Joint  Strike  Fighter’s  critical  technologies  are 
projected  to  mature  to  low  technical  risk  at  the  start  of  the  engineering 
and  manufacturing  development  phase,  we  used  the  technology  readiness 
level  tool  and  information  provided  by  Joint  Strike  Fighter  program 
officials  and  contractor  officials  at  the  Boeing  Company,  Seattle, 
Washington;  Lockheed  Martin  Aeronautics  Company,  Fort  Worth,  Texas; 
and  Pratt  &  Whitney,  East  Hartford,  Connecticut.  During  our  previous 
review,  we  had  obtained  detailed  briefings  from  Boeing  and  Lockheed 
Martin  officials  on  their  plans  to  mature  critical  technologies  prior  to  the 
date  for  awarding  the  engineering  and  manufacturing  development 
contract,  then  scheduled  for  April  2001.  We  had  also  obtained  program 
office  and  contractor  assessments  of  the  expected  technology  readiness 
levels  for  the  critical  technologies  at  April  2001.  During  our  current  review, 
we  obtained  detailed  briefings  from  program  office  personnel  on  the 
status  of  critical  technologies.  We  also  obtained  detailed  briefings  from 
Boeing,  Lockheed  Martin,  and  Pratt  &  Whitney  officials  on  the  contractors’ 
progress  in  maturing  critical  technologies  and  any  further  maturation 
plans  through  October  2001.  We  compared  the  latest  information  from  the 
program  office  and  the  contractors  to  the  information  obtained  during  our 
prior  review  to  determine  if  the  critical  technologies  had  been  matured  to 
higher  technology  readiness  levels  and  the  levels  achieved. 

We  conducted  our  review  from  April  through  September  2001  in 
accordance  with  generally  accepted  government  auditing  standards. 


We  are  sending  copies  of  this  report  to  the  congressional  defense 
committees;  the  Honorable  Donald  H.  Rumsfeld,  Secretary  of  Defense;  the 
Honorable  James  G.  Roche,  Secretary  of  the  Air  Force;  the  Honorable 
Gordon  R.  England,  Secretary  of  the  Navy;  General  James  L.  Jones, 
Commandant  of  the  Marine  Corps;  and  the  Honorable  Mitchell  E.  Daniels, 
Jr.,  Director,  Office  of  Management  and  Budget.  We  will  also  make  copies 
available  to  other  interested  parties  on  request. 
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Please  contact  me  at  (202)  512-4841  if  you  or  your  staff  have  any  questions 
concerning  this  report.  Key  contributors  to  this  report  were  Robert 
Pelletier  and  Brian  Mullins. 

Sincerely  yours, 


Katherine  V.  Schinasi 
Director,  Acquisition  and 
Sourcing  Management 
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Appendix  I:  Technology  Readiness  Levels  and 
Their  Descriptions 


Technology  readiness  level 

Description 

1.  Basic  principles  observed  and 
reported. 

Lowest  level  of  technology  readiness.  Scientific  research  begins  to  be  translated  into 
applied  research  and  development.  Examples  might  Include  paper  studies  of  a 
technology’s  basic  properties. 

2.  Technology  concept  and/or  application 
formulated. 

Invention  begins.  Once  basic  principles  are  observed,  practical  applications  can  be 
invented.  The  application  is  speculative  and  there  is  no  proof  or  detailed  analysis  to 
support  the  assumption.  Examples  are  still  limited  to  paper  studies. 

3.  Analytical  and  experimental  critical 
function  and/or  characteristic  proof  of 
concept. 

Active  research  and  development  is  Initiated.  This  includes  analytical  studies  and 
laboratory  studies  to  physically  validate  analytical  predictions  of  separate  elements  of  the 
technology.  Examples  include  components  that  are  not  yet  integrated  or  representative. 

4.  Component  and/or  breadboard 
validation  in  laboratory  environment. 

Basic  technological  components  are  Integrated  to  establish  that  the  pieces  will  work 
together.  This  is  relatively  “low  fidelity”  compared  to  the  eventual  system.  Examples 

Include  integration  of  “ad  hoc”  hardware  in  a  laboratory. 

5.  Component  and/or  breadboard 
validation  in  relevant  environment. 

Fidelity  of  breadboard  technology  increases  significantly.  The  basic  technological 
components  are  integrated  with  reasonably  realistic  supporting  elements  so  that  the 
technology  can  be  tested  in  a  simulated  environment.  Examples  include  “high  fidelity” 
laboratory  integration  of  components. 

6.  System/subsystem  model  or  prototype 
demonstration  in  a  relevant  environment. 

Representative  model  or  prototype  system,  which  is  well  beyond  the  breadboard  tested 
for  TRL  5,  is  tested  In  a  relevant  environment.  Represents  a  major  step  up  in  a 
technology’s  demonstrated  readiness.  Examples  include  testing  a  prototype  in  a  high 
fidelity  laboratory  environment  or  in  simulated  operational  environment. 

7.  System  prototype  demonstration  in  an 
operational  environment. 

Prototype  near  or  at  planned  operational  system.  Represents  a  major  step  up  from  TRL  6, 
requiring  the  demonstration  of  an  actual  system  prototype  in  an  operational  environment, 
such  as  in  an  aircraft,  vehicle,  or  space.  Examples  include  testing  the  prototype  in  a  test 
bed  aircraft. 

8.  Actual  system  completed  and  “flight 
qualified”  through  test  and 
demonstration. 

Technology  has  been  proven  to  work  in  its  final  form  and  under  expected  conditions.  In 
almost  all  cases,  this  TRL  represents  the  end  of  true  system  development.  Examples 

Include  developmental  test  and  evaluation  of  the  system  in  Its  Intended  weapon  system  to 
determine  if  it  meets  design  specifications. 

9.  Actual  system  “flight  proven”  through 
successful  mission  operations. 

Actual  application  of  the  technology  in  its  final  form  and  under  mission  conditions,  such  as 
those  encountered  In  operational  test  and  evaluation.  In  almost  all  cases,  this  is  the  end 
of  the  last  “bug  fixing”  aspects  of  true  system  development.  Examples  include  using  the 
system  under  operational  mission  conditions. 

Source;  Best  Practices:  Better  Management  of  Technology  Development  Can  Improve  Weapon 
System  Outcomes  {GAO/HS\ AD-99-1 62,  July  30,  1999). 
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Appendix  II: 
Descriptions 

Critical  Technologies  and  Their 

Critical  technologies 

Description 

Prognostics  and  health  management 

Involves  the  ability  to  detect  and  isolate  the  cause  of  aircraft  problems  and  then  predict 
when  maintenance  activity  will  have  to  occur  on  systems  with  pending  failures.  Life-cycle 
cost  savings  are  dependent  on  prognostics  and  health  management  through  improved 
sortie  generation  rate,  reduced  logistics  and  manpower  requirements,  and  more  efficient 
inventory  control. 

Integrated  flight  propulsion  control 

Includes  integration  of  propulsion,  vehicle  management  system,  and  other  subsystems 
as  they  affect  aircraft  stability,  control,  and  flying  qualities  (especially  short  take-off  and 
vertical  landing).  Aircraft  improvements  are  to  reduce  pilot  workload  and  Increase  flight 
safety. 

Subsystems 

Includes  areas  of  electrical  power,  electrical  wiring,  environmental  control  systems,  fire 
protection,  fuel  systems,  hydraulics,  landing  gear  systems,  mechanisms,  and  secondary 
power.  Important  for  reducing  aircraft  weight,  decreasing  maintenance  cost,  and 
improving  reliability. 

Integrated  support  systems 

Involves  designing  an  integrated  support  concept  that  includes  an  aircraft  with 
supportable  stealth  characteristics  and  improved  logistics  and  maintenance  functions. 
Life-cycle  cost  savings  are  expected  from  improved  low  observable  maintenance 
techniques  and  streamlined  logistics  and  inventory  systems. 

Integrated  core  processor 

Includes  the  ability  to  use  commercial-based  processors  in  an  open  architecture  design 
to  provide  processing  capability  for  radar,  information  management,  communications, 
etc.  Use  of  commercial  processors  reduces  development  and  production  costs  and  an 
open  architecture  design  reduces  future  development  and  upgrade  costs. 

Radar 

Includes  advanced  integration  of  communication,  navigation,  and  Identification  functions 
and  electronic  warfare  functions  through  improved  apertures,  antennas,  modules, 
radomes,  etc.  Important  for  reducing  avionics  cost  and  weight,  and  decreasing 
maintenance  cost  through  improved  reliability. 

Manufacturing 

Involves  lean,  automated,  highly  efficient  aircraft  fabrication  and  assembly  techniques. 
Manufacturing  costs  should  be  less  through  Improved  flow  time,  lower  manpower 
requirements,  and  reduced  tooling  cost. 

Mission  systems  Integration 

involves  decreasing  pilot  workload  by  providing  information  for  targeting,  situational 
awareness,  and  survivability  through  fusion  of  radar,  electronic  warfare,  and 
communication,  navigation,  and  identification  data.  Improvements  are  achieved  through 
highly  Integrated  concept  of  shared  and  managed  resources,  which  reduces  production 
costs,  aircraft  weight,  and  volume  requirements,  in  addition  to  improved  reliability. 

Source:  Joint  Strike  Fighter  program  office. 
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Appendix  III:  Comments  From  the 
Department  of  Defense 


ACQUiSiTION. 
TECHNOLOGY 
AND  LOGISTICS 


OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 

3000  DEFENSE  PENTAGON 
WASHfNGTON.  DC  2030^3000 


1  ?  OCT 


Mrs.  Katherine  V.  Schinasi 
Director,  Acquisition  and  Sourcing  Management 
National  Security  and  International 
Affairs  Division 

U.  S.  General  Accounting  Office 
Washington,  DC  20548 

Deiir  Mrs.  Schinasi: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  General  Accounting 
Office  (GAO)  draft  report  G AO-02-39,  ‘‘JOINT  STRIKE  FIGHTER  ACQUISITION: 
Mature  Critical  Technologies  Needed  to  Reduce  Risk,”  dated  October  2001  (GAO  Code 
I20050/OSD  Case  02-39).  The  DoD  appreciates  the  oppoitunity  to  comment  on  the  draft 
report. 


The  Department  partially  concurs  with  the  GAO’s  recommendation.  The  Joint 
Sirike  Fighter  (JSF)  program  has  extensively  matured  the  critical  technologies  requited 
for  entry  into  Systems  Development  and  Demonstration  (SD&D),  which  replaces  the 
Engineering  and  Manufacturing  Development  phase  in  the  new  DoD  5000  series.  The 
Department’s  Deputy  Under  Secretaiy  of  Defense,  Science  and  Technology,  conducted  an 
independent  Technology  Readiness  Assessment  that  concluded  that  both  of  the 
competing  contractor  teams  had  matured  their  technologies  to  a  level  consistent  with 
entry  into  the  SD&D  phase.  Comments  on  the  GAO  recommendation  are  enclosed. 

The  JSF  Program  Office  has  implemented  a  rigorous  risk  management  program 
that  will  continue  to  monitor  and  addresses  technology  risks,  as  well  as  other  risks, 
throughout  the  program's  life. 


Sincerely, 


George  R.  Schneiter 
Director 

Strategic  and  Tactical  Systems 


Enclosure 


Page  17 


GAO-02-39  Joint  Strike  Fighter  Acquisition 


Appendix  ni:  Comments  From  the 
Department  of  Defense 


GAO  DRAFF  REPORT.  Dated  October  2001 
(GAO  Code:  GAO-02-39) 

"JOIN^r  STRIKE  FIGHTER  ACQUISITION:  Mature  Critical  Technologies  Needed  to 

Reduce  Risks” 

POD  COMMENTS  ON  THE  GAO  RECOMMENDATION 

RECOMMENDATION :  The  GAO  recommended  that  the  Secretory  of  Defense  delay  the 
start  of  engineering  and  manufacturing  development  until  critical  technologies  are 
matured  to  acceptable  levels.  Altemati  vely,  if  the  Secretary  of  Defense  decides  to  accept 
these  risks  and  move  the  program  into  engineering  and  manufacturing  development  as 
scheduled.  GAO  recommended  that  the  Secretaiy'  dedicate  the  resources  to  ensure  that 
maturity  of  the  critical  technologies  is  demonstrated  by  the  critical  design  review  or  defer 
the  inclusion  of  immature  technologies  from  the  approved  design. 

POD  RESPONSE:  Partially  concur. 

The  Deputy  Under  Secretary  of  Defense,  Science  and  Technology  (DUSD(S&T)), 
conducted  an  independent  technology  readiness  assessment  per  Department  of  Defense 
Instruction  5000.2,  ’’Operation  of  the  Defense  Acquisition  System,*'  October  23,  2000. 
Although  some  subcategoiies  of  the  eight  critical  technology  areas  were  rated  other  than 
low  risk  (depending  on  the  contractor),  DU$D(S&T)  concluded  that  the  technology 
maturity  of  the  JSF  program  is  sufficient  to  warrant  entry  into  the  Systems  Development 
and  Demonstration  (SD&D)  phase.  These  areas  are  already  budgeted  to  receive  the 
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Appendix  III:  Comments  From  the 
Department  of  Defense 


resources  required  during  the  SD&D  phase,  as  they  address  the  integration  of 
technologies,  which  is  a  focus  of  that  phase, 

GAG'S  primaiy  argument  in  this  report,  and  in  previous  reports  and  discussions,  is 
that  the  approach  used  for  measuring  technology  maturity  on  JSF  (Willoughby 
Templates)  is  not  an  industry'  best  practice,  GAO  would  prefer  that  the  program  use 
Technical  Readiness  Levels  (TRLs),  As  pointed  out  in  the  Department‘s  letter  to  GAO 
dated  May  10,  2001 ,  we  agree  that  an  assessment  of  technology  readiness  is  a  key  factor 
in  determining  overall  program  risk  and  readiness  to  move  into  SD&D.  However,  the 
Department  believes  that  there  are  several  tools  that  can  be  used  to  assess  technological 
maturity,  including  TRLs  and  Willoughby  Templates,  The  Department’s  support  for  use 
of  altemaiive  methodologies  is  clearly  established  in  Department  of  Defense  Regulation 
5000.2-R,  ‘'Mandator\'  Procedures  for  Major  Defense  Acquisition  Programs  and  Major 
Automated  Information  System  Acquisition  Programs,”  June  10, 2001,  which  states, 
“[djecision  authorities  shall  consider  the  recommended  TRLs  (or  some  equivalent 
methodology,  e.g,,  Willoughby  templates)  when  assessing  program  risk.”  The  JSF 
program  office  and  the  prime  contractors  used  the  Willoughby  Templates,  with 
USD(AT&L)  concunence,  throughout  the  Concept  Development  Phase,  and  will 
continue  to  use  Willoughby  Templates  for  effectively  managing  risk  on  JSF, 

JSF  has  been  consistent  in  implementation  of  its  acquisition  strategy  with  regard  to 
risk  management  from  program  inception.  Risks  have  been  identified,  baselined,  and 
tracked,  documenting  the  specific  events  required  for  reducing  the  risk  of  these  critical 
technologies,  processes,  and  system  characteristics  to  an  adequate  level  prior  to  the 
beginning  of  SD&D. 
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GAO’s  Mission 

The  General  Accounting  Office,  the  investigative  arm  of  Congress,  exists  to 
support  Congress  in  meeting  its  constitutional  responsibilities  and  to  help 
improve  the  performance  and  accountability  of  the  federal  government  for  the 
American  people,  GAO  examines  the  use  of  public  funds;  evaluates  federal 
programs  and  policies;  and  provides  analyses,  recommendations,  and  other 
assistance  to  help  Congress  make  informed  oversight,  policy,  and  funding 
decisions.  GAO’s  commitment  to  good  government  is  reflected  in  its  core  values 
of  accountability,  integrity,  and  rehability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

■  - - 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  is  through  the 
Internet.  GAO’s  Web  site  (www.gao.gov)  contains  abstracts  and  full-text  files  of 
current  reports  and  testimony  and  an  expanding  archive  of  older  products.  The 
Web  site  features  a  search  engine  to  help  you  locate  documents  using  key  words 
and  phrases.  You  can  print  these  documents  in  their  entirety,  including  charts  and 
other  graphics. 

Each  day,  GAO  issues  a  list  of  newly  released  reports,  testimony,  and 
correspondence.  GAO  posts  this  hst,  known  as  “Today’s  Reports,”  on  its  Web  site 
daily.  The  list  contains  links  to  the  fuU-text  document  files.  To  have  GAO  E-mail 
this  list  to  you  every  afternoon,  go  to  our  home  page  and  complete  the  easy-to-use 
electronic  order  form  found  under  “To  Order  GAO  Products.” 

Order  by  Mail  or  Phone 

The  first  copy  of  each  printed  report  is  free.  Additional  copies  are  $2  each.  A 
check  or  money  order  should  be  made  out  to  the  Superintendent  of  Documents. 
GAO  also  accepts  VISA  and  Mastercard.  Orders  for  100  or  more  copies  mailed  to  a 
single  address  are  discounted  25  percent.  Orders  should  be  sent  to: 

U.S.  General  Accounting  Office 

P.O.  Box  37050 

Washington,  D.C.  20013 

To  order  by  phone:  Voice:  (202)  512-6000 

TDD:  (301)  413-0006 

Fax:  (202)  258-4066 

Visit  GAO’s  Document 
Distribution  Center 

GAO  Building 

Room  1100,  700  4th  Street,  NW  (comer  of  4th  and  G  Streets,  NW) 

Washington,  D.C.  20013 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

■ 

Contact: 

Web  site:  ww-w.gao.gov/fraudnet/fraudnet.htm, 

E-mail:  fraudnet@gao.gov,  or 

1-800-424-5454  (automated  answering  system). 

Public  Affairs 

Jeff  Nelligan,  Managing  Director,  NelliganJ@gao.gov  (202)  512-4800 

U.S.  General  Accounting  Office,  441  G.  Street  NW,  Room  7149, 

Washington,  D.C.  20548 
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